services/apps/mailing_list_integration/src/crowdmail/worker/list_worker.py:90

• Initial onboarding materializes every commit ID and retains every parsed DB/Kafka record until the entire archive finishes. For LKML-scale archives this is unbounded memory usage and no progress is checkpointed before a likely OOM/restart. Process bounded commit batches and persist each batch/head checkpoint before continuing, similar to the 250-commit chunking in git_integration/src/crowdgit/services/commit/commit_service.py:709-743.
  .github/workflows/backend-lint.yaml:147
• This CI job installs pytest and the PR adds a substantial test suite, but the job runs only Ruff. As a result, parser regressions can merge even though the PR reports the tests as coverage. Run pytest in this job after lint/format checks.

      - name: Check Python linting and formatting
        if: steps.changes.outputs.python_changed == 'true'
        run: |
          uv run ruff check src/ --output-format=github
          uv run ruff format --check src/

services/apps/mailing_list_integration/src/crowdmail/worker/list_worker.py:31

• This introduces a new class-based worker, contrary to the repository's functions-over-classes direction for new service code (CLAUDE.md:42-43 and services-checklist.md:41-43). Keep shutdown state in a small functional runner/context and expose plain processing functions so the polling and per-list logic remain independently testable.

services/apps/mailing_list_integration/src/crowdmail/worker/list_worker.py:84

• This ignores the stored source_url; ensure_mirror always clones https://lore.kernel.org/{name}/. A valid public-inbox URL with a different host or path will silently ingest the wrong archive. Pass the source URL to the mirror layer and keep the local directory key separate.
  services/apps/mailing_list_integration/src/crowdmail/worker/list_worker.py:91
• On initial onboarding, new_commits(..., None) materializes the shard’s entire history, and this method then retains every DB and Kafka record until all shards finish. Public-inbox shards can contain hundreds of thousands of messages, so the worker can exhaust memory before reaching the existing insert chunks. Iterate in bounded batches and checkpoint each batch only after persistence and emission succeed.
  services/apps/mailing_list_integration/src/crowdmail/worker/list_worker.py:93
• read_email launches synchronous Git subprocesses on the FastAPI event loop—twice per message. A large import will block health requests, shutdown handling, and other async work for long periods. Offload this blocking call to a thread.
  services/apps/mailing_list_integration/src/crowdmail/worker/list_worker.py:94
• Any parser exception escapes this per-commit loop, marks the whole list failed, and leaves the shard head unchanged. The next retry starts from the same malformed message, so one poison email can permanently block every later message in the shard. Catch failures per commit and skip/quarantine them with observable logging or metrics, as the Git integration does in commit_service.py:652-655.
  services/apps/mailing_list_integration/src/crowdmail/services/parse/noteren.py:72
• This helper is called by the long-running worker, so sys.exit() raises SystemExit, bypasses its except Exception handlers, and can terminate the service because of one malformed commit. Raise a domain exception and let the worker apply its per-message failure policy instead.
  services/apps/mailing_list_integration/src/crowdmail/services/parse/noteren.py:87
• The blob-read failure also calls sys.exit(). In worker mode this can terminate the entire service instead of failing one message/list. Raise a normal domain exception here, consistent with the commit-not-found branch.
  services/apps/mailing_list_integration/src/crowdmail/services/parse/noteren.py:354
• Messages without Message-ID are emitted with an empty sourceId. Activity deduplication keys include sourceId, so multiple such messages in the same segment/platform/type/channel can collapse into one activity. Use a stable fallback such as the Git commit/blob ID, or explicitly skip these messages.
  services/apps/mailing_list_integration/src/crowdmail/services/mirror/mirror_service.py:58
• The five-minute default applies to initial public-inbox-clone and fetch operations. LKML-scale archives can legitimately take much longer, so onboarding will repeatedly kill the clone and never complete. Use a separately configurable mirror timeout sized for large archives while retaining shorter limits for lightweight commands.
  services/apps/mailing_list_integration/src/crowdmail/database/crud.py:117
• A process crash after acquisition leaves state='processing' and lockedAt set. This exclusion, combined with both selectors requiring lockedAt IS NULL, means the list is never acquired again; stale onboarding rows also consume the concurrency count forever. Treat locks as leases and reclaim/reset processing rows older than a configured threshold.

    states_to_exclude = (ListState.PENDING, ListState.PROCESSING, ListState.PENDING_REONBOARD)

services/apps/mailing_list_integration/README.md:16

• This documentation still says the new worker emits platform=groupsio, while the implementation, seed, activity type, and identities all use platform=mailinglist. The same stale Groups.io references recur at lines 46, 62, and 66; update them so operators verify the actual platform.

  parses new messages, writes activities to `integration.results`, and emits
  Kafka messages to the `data-sink-worker` topic — same plumbing as
  git_integration, with `platform=groupsio`.

services/libs/data-access-layer/src/mailinglist/index.ts:76

• Re-onboarding a previously removed URL hits this conflict branch, but the update leaves deletedAt populated. The worker explicitly filters deleted lists, so the connect call succeeds while the list is never processed. Also reset listProcessing when ownership moves to a different integration; otherwise the new project inherits the old shard heads and misses the archive.
  services/libs/data-access-layer/src/mailinglist/index.ts:72
• The preceding ownership query does not make this upsert atomic. Two integrations connecting the same new sourceUrl can both observe no conflict; after one inserts, the other executes this DO UPDATE and silently steals the list. Enforce active ownership in the conflict statement itself and translate a rejected update into the existing 400 response.
  backend/src/api/integration/helpers/mailingListAuthenticate.ts:28
• sourceUrl is later passed to public-inbox-clone, so accepting any non-empty string allows an editor to make the worker access local paths or arbitrary network targets. Restrict this to supported public HTTP(S) URLs and reject loopback/private destinations (including redirects), with defense-in-depth validation in the worker.

        sourceUrl: z.string().trim().min(1),

services/apps/mailing_list_integration/src/crowdmail/database/crud.py:83

• This count check does not serialize concurrent acquisitions. For example, when two onboardings are active and the limit is three, two workers can both read count = 2, lock different pending rows via SKIP LOCKED, and raise the active count to four. Use a transaction-scoped advisory lock or a locked semaphore row around the count-and-acquire operation.

            AND c.count < $3

backend/src/api/integration/helpers/mailingListAuthenticate.ts:28

• sourceUrl is accepted as any non-empty string and later passed to public-inbox-clone from the worker. A tenant editor can therefore make the service access local schemes or internal network endpoints. Restrict this to supported public-inbox URL schemes and enforce a public-host/private-address policy before cloning.

    return false

backend/src/api/integration/helpers/mailingListAuthenticate.ts:23

• Reject duplicate sourceUrl entries at validation time. PostgreSQL raises ON CONFLICT DO UPDATE command cannot affect row a second time when the same URL appears twice in this array, turning validly shaped client input into a 500 response.

// bare non-URL string.

services/libs/data-access-layer/src/mailinglist/index.ts:76

• Re-onboarding a previously removed URL leaves deletedAt populated because the conflict update never clears it. The worker filters deleted rows, so the API reports success but the list is never processed again.
  services/libs/data-access-layer/src/mailinglist/index.ts:83
• When a soft-deleted list is assigned to a different integration/segment, this preserves the previous owner's lastProcessedHeads. The new project then starts after those heads and permanently misses the archived messages that were ingested only into the old segment. Reset processing state and heads when ownership changes, while preserving them for same-integration updates.
  services/apps/mailing_list_integration/src/crowdmail/database/crud.py:83
• The onboarding cap is not atomic. Concurrent workers can read the same c.count < MAX_CONCURRENT_ONBOARDINGS snapshot, lock different pending rows via SKIP LOCKED, and both exceed the configured limit. Serialize the count-and-acquire decision (for example with a transaction advisory lock or a dedicated semaphore row).

            AND c.count < $3

services/apps/mailing_list_integration/src/crowdmail/server.py:13

• The PR description says the FastAPI lifespan is tested, but the service's only test module is test_noteren.py and contains no lifespan, worker, database, mirror, or queue tests. Add coverage for startup/shutdown and worker-task cancellation, or correct the stated test coverage.

services/libs/data-access-layer/src/mailinglist/index.ts:76

• Re-onboarding a previously removed source cannot work correctly here. The conflict check intentionally ignores soft-deleted rows, but this conflict branch neither clears deletedAt nor resets listProcessing; the worker therefore filters the reclaimed row out, and simply clearing deletedAt would still preserve the previous project's shard heads and skip its archive. Resurrect the row and conditionally reset processing state when ownership/segment changes, while preserving progress only for the same integration.
  backend/src/api/integration/helpers/mailingListAuthenticate.ts:27
• Checking only the https: scheme still lets any tenant editor make the worker connect to arbitrary internal hosts or private IPs via public-inbox-clone (including through redirects), creating an SSRF path. Restrict sources to an approved public-inbox host allowlist, or resolve and reject loopback/private/link-local destinations and enforce the same policy for every redirect.

const isSafeSourceUrl = (sourceUrl: string): boolean => {
  try {
    return new URL(sourceUrl).protocol === 'https:'
  } catch {

backend/src/api/integration/helpers/mailingListAuthenticate.ts:47

• Duplicate detection compares raw strings, so equivalent archive URLs such as https://lore.kernel.org/list and https://LORE.KERNEL.ORG/list/ bypass both this check and the database uniqueness/conflict guard. Normalize the URL before validation and persistence (hostname casing, default port, trailing slash, and disallow query/fragment/userinfo) so one archive has one canonical key.

    .refine((lists) => new Set(lists.map((l) => l.sourceUrl)).size === lists.length, {
      message: 'lists contains duplicate sourceUrl entries',
    }),

.github/workflows/backend-lint.yaml:123

• Dependency-only changes to pyproject.toml or uv.lock make this job report success without installing dependencies or running tests because this condition only matches .py files. Include those project files in the change predicate so a broken lockfile or dependency update cannot bypass CI.

          if git diff --name-only ${{ github.event.pull_request.base.sha }} ${{ github.sha }} | grep -q "^services/apps/mailing_list_integration/.*\.py$"; then

services/apps/mailing_list_integration/pyproject.toml:10

• The package metadata points to the repository's Apache-2.0 LICENSE, but the imported noteren.py module declares GPL-2.0-or-later, so the built/distributed crowdmail package has conflicting license declarations. Resolve the licensing for the combined package (or isolate the GPL component) and make the package metadata/notices match the resulting distribution terms.

license = { file = "LICENSE" }

services/apps/mailing_list_integration/src/crowdmail/worker/list_worker.py:160

• The new ingestion core has no tests for batching/checkpoint order or DB/Kafka failure recovery; the only test module exercises the parser. Add worker tests proving heads advance only after a successful Kafka send, skipped messages still advance, and failed sends remain retryable without duplicate result rows.

services/apps/mailing_list_integration/src/crowdmail/database/crud.py:86

• The onboarding limit is not atomic. Multiple workers can read the same current_onboarding_count, lock different pending rows with SKIP LOCKED, and all pass c.count < $3, exceeding MAX_CONCURRENT_ONBOARDINGS. Serialize the count-and-acquire decision (for example with an advisory/sentinel lock) or use a permit model so concurrent clone load is actually bounded.

            AND l."deletedAt" IS NULL
            AND c.count < $3
        ORDER BY lp.priority ASC, lp."createdAt" ASC
        LIMIT 1
        FOR UPDATE OF lp SKIP LOCKED

services/libs/data-access-layer/src/mailinglist/index.ts:84

• Reclaiming a soft-deleted source under another integration preserves its old listProcessing row because this conflict path does nothing. The ownership check intentionally permits deleted rows, while the upsert changes integrationId; consequently the new project starts from the previous project's heads and never receives historical messages. Reset state, timestamps, and lastProcessedHeads when ownership changes.
  backend/src/api/integration/helpers/mailingListAuthenticate.ts:29
• Restricting the scheme to HTTPS does not prevent SSRF: a tenant editor can still target private/link-local hosts with TLS, and redirects may reach a different host. The worker then performs the server-side fetch. Apply the repository's network URL validation or enforce an explicit public-inbox host allowlist, including redirect validation.

const isSafeSourceUrl = (sourceUrl: string): boolean => {
  try {
    return new URL(sourceUrl).protocol === 'https:'
  } catch {
    return false
  }

backend/src/api/integration/helpers/mailingListAuthenticate.ts:41

• sourceUrl is persisted and compared byte-for-byte, but the worker strips trailing slashes before cloning. Equivalent inputs such as https://lore.kernel.org/list and https://lore.kernel.org/list/ therefore bypass both duplicate validation and cross-integration ownership checks, creating duplicate ingestion of the same archive. Canonicalize URLs before they reach the service/DAL and enforce uniqueness on that canonical value.

        sourceUrl: z.string().trim().min(1).refine(isSafeSourceUrl, {
          message: 'sourceUrl must be a valid https:// URL',
        }),

backend/src/api/integration/helpers/mailingListAuthenticate.ts:26

• Checking only the scheme still permits authenticated callers to make the worker clone attacker-chosen internal HTTPS hosts (including loopback/private DNS targets), creating a blind SSRF from the worker network. Given the initial hardcoded deployment, allowlisting approved public-inbox hosts is safest; otherwise resolve and reject private/link-local/loopback addresses in the worker and re-check every redirect before fetching.

const isSafeSourceUrl = (sourceUrl: string): boolean => {
  try {
    return new URL(sourceUrl).protocol === 'https:'

services/libs/data-access-layer/src/mailinglist/index.ts:75

• The preflight ownership query does not make this upsert atomic. Two projects connecting the same previously absent source concurrently can both observe no conflict; the transaction that loses the insert race then executes this unconditional update and silently reassigns the first project's list. Make the conflict update conditional on the existing owner being this integration (or the row being deleted), and surface a conflict when no row is returned.
  services/libs/data-access-layer/src/mailinglist/index.ts:84
• When a soft-deleted source is connected by a different integration or segment, the list row is re-pointed but this leaves its old processing row and lastProcessedHeads intact. The new project will therefore skip the entire archive up to the previous owner's checkpoint and only receive future messages. Reset processing state when ownership/segment changes, while preserving progress for ordinary updates by the same owner.
  services/apps/mailing_list_integration/src/crowdmail/database/crud.py:86
• This count check is not serialized with acquiring a row. With multiple worker replicas, several transactions can read the same count below the limit, lock different pending rows, and all proceed, exceeding MAX_CONCURRENT_ONBOARDINGS—the only scenario where a limit above one matters because each process handles one list at a time. Use a transactional semaphore/advisory lock or another atomic capacity mechanism.

            AND l."deletedAt" IS NULL
            AND c.count < $3
        ORDER BY lp.priority ASC, lp."createdAt" ASC
        LIMIT 1

services/apps/mailing_list_integration/src/crowdmail/database/crud.py:190

• Stale-reclaim can deliberately let two workers process one list, but this checkpoint is not tied to the lease acquired by the caller. A reclaimed worker can later overwrite the same shard with an older SHA; its subsequent mark/release can also overwrite the new worker's state and clear its lock. Carry an acquisition token (for example the acquired lockedAt value or a lease UUID) and require it in every checkpoint, mark, and release WHERE clause.

    if not mailing_list:
        mailing_list = await acquire_recurrent_list()
    return mailing_list

services/apps/mailing_list_integration/src/crowdmail/services/parse/noteren.py:423

• For a range containing more than one commit, each call prints a separate indented JSON document back-to-back, so the advertised output is neither valid JSON nor valid JSON Lines. Emit one JSON array for the range, or emit one compact JSON object per line and document the output as JSONL.
  services/apps/mailing_list_integration/src/crowdmail/worker/list_worker.py:159
• The tests added in this service exercise only the parser/CLI; the core flush sequence has no coverage. Please add async worker tests for DB-insert failure, Kafka failure, checkpoint failure, and skipped-message behavior to verify that retries neither lose messages nor advance heads prematurely. This state machine is the reliability boundary of the integration.
  services/apps/mailing_list_integration/src/crowdmail/server.py:13
• The PR description says the server import and FastAPI lifespan are tested, but the only test module under this service is src/test/test_noteren.py, which never imports crowdmail.server or exercises this lifespan. Add the claimed lifecycle test (including worker startup and timeout/cancellation shutdown), or correct the test-coverage statement.

backend/src/api/integration/helpers/mailingListAuthenticate.ts:26

• This validation still permits SSRF targets such as https://127.0.0.1, private RFC1918 addresses, link-local metadata endpoints, and hostnames that resolve internally. The worker later passes this tenant-controlled URL to public-inbox-clone from the service network, so HTTPS alone does not make the fetch safe. Restrict sources to an approved public-inbox host set, or enforce DNS/IP checks (including redirects and DNS rebinding) in the worker before fetching.

const isSafeSourceUrl = (sourceUrl: string): boolean => {
  try {
    return new URL(sourceUrl).protocol === 'https:'

services/libs/data-access-layer/src/mailinglist/index.ts:76

• The ownership precheck is not atomic with this upsert. Two integrations can both observe no conflict, then the second ON CONFLICT branch silently overwrites the first integration's segmentId/integrationId, defeating the stated ownership protection. Make ownership enforcement part of the write (for example, lock the source row or use a conditional conflict update) and surface a conflict when the atomic write does not acquire ownership.
  services/apps/mailing_list_integration/src/crowdmail/database/crud.py:211
• Merging JSON does not prevent a stale worker from moving the same shard backward. A live run can exceed the 4/12-hour reclaim timeout, allowing a second worker to start from its checkpoint; either worker can then overwrite that shard key, and stale runs can also mark/release state owned by the newer run. Add a renewable lease/token and make checkpoints, completion, and release conditional on still owning that lease (or otherwise compare shard progress monotonically).

        SET "lastProcessedHeads" = "lastProcessedHeads" || $1::jsonb,